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STEREOSCOPIC DISPLAY SYSTEM WITH FLEXIBLE RENDERING 
OF DISPARITY MAP ACCORDING TO THE STEREOSCOPIC FUSING 
CAPABILITY OF THE OBSERVER 

FIELD OF THE INVENTION 

The invention relates generally to the field of displays, and in 
particular to stereoscopic display systems. More specifically, the invention relates 
to a stereoscopic display system that provides a customized rendering for groups 
of similar individuals or each individual user. 

BACKGROUND OF THE INVENTION 

The normal human visual system provides two separate views of 
the world through our two eyes. Each eye has a horizontal field of view of about 
60 degrees on the nasal side and 90 degrees on the temporal side. A person with 
two eyes, not only has an overall broader field of view, but also has two slightly 
different images formed at her two retinas, thus forming different viewing 
perspectives. In normal human binocular vision, the disparity between the two 
views of each object is used as a cue by the human brain to derive the relative 
depth between objects. This derivation is accomplished by comparing the relative 
horizontal displacement of corresponding objects in the two images. 

Stereoscopic displays are designed to provide the visual system 
with the horizontal disparity cue by displaying a different image to each eye. 
Known stereoscopic displays typically display a different image to each of the 
observers' two eyes by separating them in time, wavelength or space. These 
systems include using liquid crystal shutters to separate the two images in time, 
lenticular screens, barrier screens or auto-stereoscopic projection to separate the 
two images in space, and the use of color filters or polarizers to separate the two 
images based on optical properties. 

Unfortunately, the visual information provided by a stereoscopic 
display differs in many ways from viewing real world objects. This difference in 
visual information or visual cues often leads to user discomfort or the inability to 
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fuse the two images (i.e., to perceive the two images displayed to the two eyes as 
one fused image rather than two separate images). One of the important 
differences between the visual information provided by most prior art stereoscopic 
display systems and the real world environment arises from the fact that the user 
5 of these displays must accommodate, or focus, at a single plane in space when 
viewing a stereoscopic display while the disparity cues that are provided indicate 
that the objects are at different planes in space. This differs from the real world 
visual environment where the accommodative and disparity cues provide 
consistent information. This is a very important difference since the vergence of 

1 0 our eyes and their accommodation distance are reflexively linked, often causing 
accommodation distance to track convergence distance as the two eyes converge 
to place an important object onto the fovea of each eye. 

Another important difference is that stereoscopic display systems 
have the ability to augment the disparity cues, providing larger amounts of 

1 5 disparity information than actually exists in the real world. This provision can 
provide a greater sense of depth and greater perceived separation of objects in an 
image. This can be particularly important in detection or identification tasks, 
where it is important to separate a target (e.g., a cancerous lesion, a surveillance 
threat, or a potential rescue victim) from a complex background. Unfortunately, 

20 when this disparity becomes too large, users of stereoscopic display systems often 
experience discomfort. 

It is well understood that there is an upper limit for how much 
disparity the human brain can fuse (see Binocular Vision and Stereopsis , Chapter 
entitled "The limits of stereoscopic vision" by Howard and Rogers, p. 159, 1995). 

25 It is also well understood that some individuals are not able to interpret 

stereoscopic information and therefore are unable to benefit from the cues 
provided by a stereoscopic display system. It is understood to a lesser degree, 
however, that there is significant individual variability in observers' ability to 
comfortably fuse two images that have horizontal disparity. 

30 In the field of psychology, it has been demonstrated that the 

fusional range that is common to all users is much smaller than that for some 
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individual users (see ref. "Image distortions in stereoscopic video systems" 
Proceedings of the SPIE, Volume 1915, Stereoscopic Displays and Applications 
IV, by Andrew Woods, Tom Docherty, and Rolf Koch, pp. 1-13, 1993). This 
research finding is supported by research in the field of optometry, where a 
5 method of quantifying the range of convergent and divergent angles that each 
individual is able to fuse has been applied to understand an individual's visual 
performance. It is well understood that this range of convergent and divergent 
angles that can be fused, referred to as an individual's fusional reserve, differs 
from individual to individual and differs with differences in the accommodative 

10 stimulus that is provided. Importantly, research in this field has shown that some 
individuals can fuse a large range of convergent angles and a small range of 
divergent angles while other individuals can fuse larger ranges of divergent angles 
than convergent angles. Because of this individual variability, if a stereoscopic 
system is developed to provide comfortable stereoscopic images to all users, the 

1 5 range of disparity that can be shown will be extremely limited. 

In the field of stereoscopic display it has been realized that a 
stereoscopic display system cannot provide high quality, comfortable horizontal 
disparity information unless the system takes into account the limitation set by the 
human binocular system. U.S. Pat. No. 4,647,965 describes a stereoscopic 

20 imaging system that has the ability to adjust the horizontal positions of the 

corresponding objects in the stereoscopic pair to provide quality and viewing 
comfort for viewing three-dimensional content. The system consists of image 
capture, image digitization, image processing, and digital to analog conversion. 
The image-processing algorithm identifies corresponding objects in the 

25 stereoscopic pair, and shifts the objects in one of the two images to reduce the 

disparities between the views. However, this system does not provide a means for 
determining an individual's ability or for clustering an individual into a group of 
individuals that may have similar visual performance. Further, image display as 
described in this display is performed through analog media that can not be altered 

30 for each individual and therefore the images produced by this system cannot be 
rendered for users of different ability and therefore to be successful this system 



must reduce the maximum disparity to the small range that all observers can 
comfortably fuse or accept the fact that some percentage of users will experience 
discomfort while viewing these stereoscopic images. 

U.S. Pat. No. 6,175,379B1 discusses a computer graphics driven 
5 stereoscopic display system which also provides a customized rendering for 

different viewing conditions. Specifically, the system alters the rendering camera 
separation, focal length, and/or distance from objects in the scene as a function of 
the viewing conditions that include the display size, viewing distance, and 
resolution. The intent of this patent is to render stereoscopic information that can 

1 0 be comfortably viewed by all users. As described in this patent, computer 

graphics experts decide on the acceptable range of disparities that can be fused by 
all users when rendering the computer graphics scene. Image content outside the 
fusional area are clipped from view or their contrast and/or focus are reduced to 
make them unusable. Once again, this system does not provide a means for 

1 5 determining an individual's ability and the output of this system uses analog 

media that cannot be altered for each individual user who has a different ability. 

User comfort has also been considered in commercial products 
such as the NVIDIA 3D stereo driver. This driver can generate and display a 
stereoscopic image pair from one computer generated image with embedded depth 

20 information about the computer generated graphic objects. This driver comes with 
a number of controls for use in stereo game playing, including stereo separation 
adjustment to suit individual adaptation to stereo viewing, hot keys for in-game 
stereo control, stereo test process for determining the best screen mode to use, and 
custom game configurations that tailor the stereo process to a wide range of 

25 games. This product provides the technical capability to adjust the range of 
disparity for a range of stereoscopic scenes. However, this system does not 
provide a method to link the capability of individual users to fuse stereoscopic 
images to the stereoscopic rendering process. As a result, the viewing comfort can 
only be achieved by the dynamic adjustment during viewing. Considering that in 

30 a game-playing environment the range of disparity may vary significantly from 

scene to scene, the dynamic nature of the adjustment can be a challenging task for 



the users. Furthermore, the optimization between viewing comfort and good 
stereoscopic perception cannot be achieved by a method of dynamic adjustment 
such as the one provided by NVIDIA as it does not consider the actual range of 
disparities present in any scene. 
5 There is a need, therefore, for creating an adaptive stereoscopic 

display system that can obtain and utilize the knowledge of the stereo capability of 
the user to enhance the rendering of stereo images. 

SUMMARY OF THE INVENTION 

1 0 The present invention is directed to overcoming one or more of the 

problems set forth above. According to one aspect of the present invention, a 
method is provided for customizing scene content, according to a user or a cluster 
of users, for a given stereoscopic display, including obtaining customization 
information about the user; obtaining a scene disparity map for a pair of given 

15 stereo images and/or a three-dimensional (3D) computer graphic model; and 
determining an aim disparity range for the user. The method of the present 
invention also generates a customized disparity map and/or rendering conditions 
for a three-dimensional (3D) computer graphic model correlating with the user's 
fusing capability of the given stereoscopic display; and re-renders the stereo 

20 images for subsequent display. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The above and other objects, features, and advantages of the 
present invention will become more apparent when taken in conjunction with the 
25 following description and drawings wherein identical reference numerals have 
been used, where possible, to designate identical features that are common to the 
figures, and wherein: 

Fig. 1 is a diagram of the system employed in the practice of the 
present invention. 

30 Fig. 2 is a flow chart showing the steps of the method of the present 

invention. 
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Fig. 3 shows an example user interface screen for inputting 
rendering intent. 

Fig. 4 is an example graphical user interface for on-display 
measurement of user stereo fusing capability. 
5 Fig. 5 is a flow chart showing the steps of determining the aim 

disparity range using customization information. 

Fig. 6 is a graphical illustration of the default range of disparity as a 
function of the viewing distance. 

Fig. 7 is a flow chart showing the steps of a method to calculate the 
10 aim disparity range based on optometric data. 

Fig. 8 is an example user interface for inputting optometric data. 

Fig. 9 is an illustration of a plot representing optometric data and 
the range of disparity that can be fused as a function of accommodation distance. 

Fig. 10 shows the relationship of the spatial arrangement of the 
15 cameras, the scene depth range, and the corresponding scene disparity range. 

Fig. 1 1 shows an illustration of remapping between scene disparity 
and aim disparity range. 

Fig. 12(a) is a plot showing one method of performing the mapping 
from input disparity to output disparity that may be employed in the practice of the 
20 present invention. 

Fig. 12(b) is a plot showing a second method of performing the 
mapping from input disparity to output disparity that may be employed in the 
practice of the present invention. 

Fig. 12(c) is a plot showing a third method of performing the 
25 mapping from input disparity to output disparity that may be employed in the 
practice of the present invention. 

To facilitate understanding, identical reference numerals have been 
used, where possible, to designate identical elements that are common to the 
figures. 

30 
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DETAILED DESCRIPTION OF THE INVENTION 

The present description is directed in particular to elements 
forming, part of, or cooperating more directly with, apparatus in accordance with 
the invention. It is to be understood that elements not specifically shown or 
5 described may take various forms well known to those skilled in the art. 

The present invention is directed towards a stereoscopic imaging 
system in which display attributes and knowledge of user's binocular visual 
performance are applied to improve the comfort and the quality of the viewing 
experience. This invention is based on the research results by the authors where a 

1 0 link between the optometric measurement and the stereo fusing capability is 

established. This system utilizes knowledge of the display attributes and user's 
binocular visual performance to determine rendering parameters for the 
stereoscopic information. These rendering parameters are used to provide a 
customized presentation of information on the display device for a group or a 

1 5 specific human observer. These rendering parameters may be used to select 
among various segments of pre-rendered content or are used to render left and 
right eye views to the observer in a way that improves the viewing experience. 
This improvement in viewing experience will often result in increased user 
comfort or enhancing the quality of the viewing experience in terms of increasing 

20 user enjoyment, engagement or presence. This improvement may also be linked 
to the improvement in the performance of the user during the completion of a 
task. 

A system of the present invention is shown in Fig. 1 . The system 
of the present invention includes an image source 10 for obtaining stereoscopic 

25 image information or computer graphics models and textures, an optional image 
processor 20 for extracting a disparity map from the image source, a storage 
device 30 for storing display attributes, a storage device 40 for storing 
customization information, a rendering processor 50 for generating a customized 
disparity map, and rendering the stereoscopic information based on this map, and 

30 a stereoscopic display device 60 for displaying the rendered stereoscopic pair of 
images. This system can optionally have an input device 70 that takes feedback 



from the user that may be used to assess the user's binocular visual performance, 
or to set the rendering intent, or to provide means for user interaction with the 
system during viewing (e.g. indicating the region of interest). It may also 
optionally have a sensor 80 that monitors the characteristics of the user including 
5 the head position, eye gaze direction, and accommodation state of the eye. 

The image source 10 may be any device or combination of devices 
that are capable of providing stereoscopic image information. For example, this 
image source may include a pair of still or video cameras capable of capturing the 
stereoscopic image information. Alternately, the image source 1 0 may be a server 
10 that is capable of storing one or more stereoscopic images. The image source 10 
may also consist of a memory device capable of providing definitions of a 
computer generated graphics environment and textures that can be used by the 
image processor to render a stereoscopic view of a three dimensional graphical 
environment. 

1 5 The optional image processor 20 may be any processor capable of 

performing the calculations that are necessary to determine the disparity between a 
pair of stereoscopic images that have been retrieved from the image source 10. For 
example, this processor may be any application specific integrated circuit (ASIC), 
programmable integrated circuit or general-purpose processor. The image 

20 processor 20 performs the needed calculation based on information from the 
storage device for display attributes 30 as well as information from the image 
source 10. 

The storage device for display attributes 30 may be any storage 
device that is capable of storing information regarding the characteristics of the 

25 display device. This storage device may be, for example, an electronically 

programmable read only memory (EPROM) that is embedded in the display itself 
or on a video card. Alternatively, it may be storage that is available on a local or 
network based computer disk such as a hard drive, CD. This storage device will 
store information for the display device such as the resolution, size of display, 

30 point of optical convergence, and plane of accommodative focus for retrieval and 



use. This storage device will also store viewing environment characteristics, such 
as the nominal observer viewing distance. 

The storage device for customization information 40 may be any 
memory device that is capable of storing information regarding the characteristics 
5 of the user. This storage device may be, for example, an electronically 

programmable read only memory (EPROM) that is embedded on a video card. 
Alternatively, it may be a storage device that is available on a local or network 
based computer disk such as a hard drive, compact disk, or solid-state memory. 
Alternatively this storage device will include a means to obtain the stereo 

10 capability of the user using a computer program. This storage device will store 
information for the user such as the capability of convergence/divergence, phoria, 
capability of accommodation, range of fusion. This storage device will also store 
the rendering intent, as specified by the user. The storage device 30 and 40 may be 
two separate devices, or may be the same device. 

1 5 The rendering processor 50 may be any processor capable of 

performing the calculations that are necessary to determine the customized 
disparity map for a specific user and specific scene content and to render the 
graphical or scene information using this disparity map. The calculation is based 
on the inputs from image processor 20, storage device for display attributes 30, and 

20 storage device for customization information 40, input device 70, and sensor 80. 
The rendering processor 50 and the image processor 20 may be two separate 
devices, or may be the same device. 

The stereoscopic display device 60 may be any display capable of 
providing a stereoscopic pair of images to a user. For example, the stereoscopic 

25 display device 60 may be a direct view device that presents an image at the surface 
of the display (i.e., has a point of accommodation and convergence at the plane of 
the display surface); such as a barrier screen Liquid Crystal display device, a CRT 
with liquid crystal shutters and shutter glasses, a polarized projection system with 
linearly or circular polarized glasses, a display employing lenticules, a projected 

30 auto-stereoscopic display, or any other device capable of presenting a pair of 

stereographic images to each of the left and right eyes at the surface of the display. 
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The stereoscopic display device 60 may also be a virtual image display that 
displays the image at a virtual location, having adjustable points of accommodation 
and convergence, such as an auto-stereoscopic projection display device, a 
binocular helmet-mounted display device or retinal laser projection display. 
5 The optional input device 70 may be a mouse or a keyboard that are 

capable of taking input from the user. It may also be a multi-modal device that 
takes input from the user via eye tracking, by voice, or by tactile action. 

The optional sensor 80 may be any tracking device that can detect 
the position of the head or the gaze direction of the eyes. For example, it can be 

10 two cameras that sense the head position, and hence is able to determine the 
distance between the head and the display screen. 

Fig. 2 is a flow chart depicting a method for implementing this 
invention. Customization information is obtained in step 100. Display attributes 
are obtained in step 1 10. Stereoscopic imagery or 3-D graphical models and 

15 textures are obtained in step 120. From this imagery or 3D graphical model data a 
scene disparity map is extracted in step 130. In step 135 an aim disparity range is 
determined based on the customization information and the display attributes. A 
remapping is completed in step 140 where a customized disparity map is 
generated. The stereoscopic scenes are re-rendered in step 150, and then displayed 

20 in step 160. 

The customization information obtained in step 100 includes a user 
profile and/or a rendering intent. The user profile defines the stereo fusing 
capability of an individual user as a function of the accommodation state. It also 
contains a unique user identifier. The rendering intent may include items such as 

25 the level of skill and the type of task to be performed. 

Fig. 3 shows options that may be depicted in an example user 
interface screen and presented to the user or a system administrator to input the 
rendering intent. The rendering intent includes task related options as shown in 
Fig. 3, such as "optimize detectability" 210, "provide a fun and comfortable 

30 experience" 2 1 2, or "maximize depth near the pointer" 2 1 4. The rendering intent 
may also include skill related options, such as "new user" 216 or "experienced 
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user" 218. The input device 70 may be used to select and confirm an option 
among these options. Menus of the type shown in Fig. 3 may be employed to 
determine any of the items that may be used to determine the rendering intent. 
Once the user selects an option from any of these menus, their selection may be 
5 stored in the storage device for customization information 40 to be retrieved for 
future use based on the user identifier. 

Fig. 4 depicts a graphical user interface for a software application 
that can be used to assess the user's stereo fusing capability on the same display 
system. Here the input device 70 is assumed to be a mouse. At the beginning of 

1 0 the test session, the slider marker 3 1 0 is in the middle of the slider bar 320. There 
is no horizontal disparity for the test stimulus 330, and the user should see the test 
stimulus 330 as having the same depth as the background 300. Then the user starts 
to move the slider marker 3 1 0 to the right. This movement increases the crossed 
disparity of the test stimulus 330 so that it appears to pop out towards the user 

1 5 relative to the background 300. At some point the test stimulus 330 would no 

longer be fusible. This indicates the upper limit of the Fusional Range for Crossed 
Disparity (FRCD). The user would click button 340 'Done' to record this range, 
and the system is reset to the original state, i.e. the slider marker 3 10 is back to the 
middle of the slider bar 320, and the test stimulus 330 resides in the same depth 

20 plane as the background 300. The limit of the Fusional Range for Uncrossed 

Disparity (FRUD) is recorded similarly with the slider marker 310 being moved to 
the left. 

Several alternative user interfaces may similarly be designed. For 
example, the display may show a test stimulus 330 with a larger disparity than 

25 anyone can be expected to fuse and the slider marker 3 1 0 can be used to decrease 
the disparity until the user is able to fuse the target at which time they can press the 
button 340 'Done'. In another embodiment, rather than having the user manually 
manipulate the fusional range of the test stimulus 330 by moving the slider marker 
310, the computer may automatically adjust the disparity of the target and the user 

30 may simply press the button 340 'Done' whenever they are or are not able to fuse 
the target. In yet another embodiment, the user interface may display a plurality of 
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test stimulus 330 having a range of disparities and the user may simply indicate the 
test stimulus with the largest disparity that they are able to fuse. It is also 
understood that some display systems may allow some pixels to have a different 
focal distance than other pixels and therefore some of the test stimulus 330 used 
5 within the user interface may have a different focal distances. Through showing a 
number of the user interface screens in which focal distance and disparity are 
changed independently of one another, a profile may be built which indicates the 
fusable disparity as a function of focal distance for each observer. In yet other 
embodiments of the user interface, the user may be asked to provide a rating or 

10 other number associated with their perception of their psychophysical response to 
the test stimulus 330. For example, the user may be shown one or more test 
stimuli 330 and asked to indicate their comfort when viewing this test stimuli by 
entering a number corresponding to their comfort into a data entry field and then 
pressing the button 340 'Done' to indicate that he response has been correctly 

1 5 entered. 

Fig. 5 shows the steps taken to determine the aim disparity range, as 
described in step 135 of Fig. 2. First, the system obtains a user identifier in step 
400 using input device 70. In step 41 0 the system determines if the user profile is 
existent or is new. If the user identifier is linked to an existing user profile, the 

20 user profile will be retrieved in step 420. If the user identifier is new, the system 
would offer options to obtain the user profile in step 430. There are three options 
to obtain this user profile. It can be obtained using the same display device with a 
computer program, as illustrated in Fig. 4. It can also be obtained offline using 
optometric assessment, and then the data may be input into the system. Lastly, it 

25 can be a default user profile that is stored in the storage device for customization 
information 40. Details of the latter two methods will be discussed in future 
sections. In step 440 a rendering intent is obtained from the input of the user or 
system administrator. In step 450 two factors are assigned based on the rendering 
intent: Cs for skill level, and Ct for task type. The values for Cs and Ct are 

30 commonly between 0 and 1 , but they may be larger than 1 for certain applications. 
For new users Cs is usually set to a value smaller than that for the experienced 
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users. The task factor Ct depends on the task types. Its value would be smaller if 
the rendering intent is to 'Provide a fun and comfortable experience 5 compared to 
'Optimize detectability' or 'Maximize depth near the pointer' where having nearly 
the maximum perceived separation in distance may be beneficial to the user. In 
5 step 460 a value is assigned to the adaptive factor Ca. The factor Ca is intended to 
compensate for the dynamic nature of the viewing experience. For example, the 
user may experience stereoscopically induced visual fatigue during long viewing 
periods. This fatigue may be detected by sensor 80, and Ca would be set to a 
smaller value to preserve user comfort. The detection can take many forms. For 
10 example, it can be a recording of the change in blinking rate. It can also be a 

recording of the change in the pattern of eye gaze direction. In step 470 an Aim 
Range of Crossed Disparity (ARCD) and an Aim Range of Uncrossed Disparity 
(ARUD) are determined, as shown in equations 1 and 2. The convergence point 
corresponds to a location in the scene space where the horizontal disparity is zero. 

15 

ARCD = Cs * Ct * Ca * FRCD (1) 
ARUD = Cs * Ct * Ca * FRUD (2) 



Fig.6 is a graphical illustration of how the default fusional ranges of 
20 crossed/uncrossed disparity may vary as a function of the viewing distance. 

Fig. 7 shows a method of using optometric data to determine the 
aim disparity range. In step 5 1 0 the system obtains optometric data. This data 
may be obtained in many ways, including requiring the user to input this 
information using the input device 70. Other ways of obtaining this information 
25 may include obtaining this information from another networked or removable 

storage device where the file is linked to the user identifier. The optometric data 
may include but are not limited to the following parameters: interpupillary 
distance, dissociated phoria, fusional reserves. 

Dissociated phoria is referred to as the amount by which the lines of 
30 sight of the eyes deviate from a corresponding fusion stimulus during monocular 
viewing. Fusional reserve is referred to as the amount of vergence, which is 
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required to overcome disparity introduced by placing base-in (BI) or base-out (BO) 
lenses in front of a person's eyes. Phoria, base-in, and base-out fusional reserves 
are usually measured at near and distant accommodation planes. 

In the preferred embodiment, the optometric data may include any 
5 or all of the following parameters: 

IPD - interpupillary distance 

d mar - Near accommodation plane for measuring data 

d distant - Distant accommodation plane for measuring data 

P near - Near phoria 

10 P distant - Distant phoria 

BO near - Near base-out fusional reserve 

BI „ear - Near base-in fusional reserve 

BO distant - Distant base-out fusional reserve 
BI distant - Distant base-in fusional reserve 

15 

An example of a user interface screen for providing optometric data 
is shown in Fig. 8. This user interface allows the user to enter their interpupillary 
distance 512, near accommodation plane for measuring data 514, the distant 
accommodation plane for measuring data 516, their near phoria 518, their distant 

20 phoria 520, their near base-out fusional reserve 522, their near base-in fusional 
reserve 524, their distant base-out fusional reserve 526 and their distant base-in 
fusional reserve 528. Once these values are entered, the user may dismiss this user 
interface button by pressing the 'OK' button 530. The optometric data are stored 
in the storage device for customization information 40. 

25 Returning again to Fig. 7, several equations are shown for 

calculating the aim disparity range according to the exemplary steps shown in Fig. 
7. In step 540 the optometric data measured for two accommodation planes are 
generalized for other accommodation planes. For this, one needs to convert the 
optometric data into common units (e.g., degrees of vergence angle, diopters) and 

30 calculate linear parameters describing phoria, BI and BO fusional reserve lines. 
The following formulas show how the slope (k) and intercept (c) of phoria, BO, 
and BI lines can be calculated in the case the data are converted into degrees of 
vergence angle: 
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Note that the parameter arctan(IPD/2d) in Equations 3-8 represent 
vergence angle for a corresponding accommodation plane. More specifically, this 
15 parameter represents how much the eyes must converge from parallelism to fixate 
an object. It will be referred to as Convergence Demand, i.e., CD. The 
Convergence Demand for a given display accommodation distance will be referred 

tO as CDdisplay 

The optometric data are calculated for the accommodation plane of 
20 the display ddi Sp iay, which can be obtained from the storage device for display 
attributes 30. 
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Based on values obtained in Equations 9-1 1 one can calculate in 
step 545 Phoria Demand (PDdispiay) and Fusional Reserve (FRdispiay) for the 
accommodation plane of the display display- 



PD = P - CD 

ri - J display r display display (12) 

^display (crossed) = BO display - CD display ^ 

FR dis P iay (uncrossed) = CD display - BI display ^ 



In step 550 a value for a desired Comfort Level, CL, is obtained. 

15 The comfort level can range from 0 (very uncomfortable) to 100 (very 

comfortable) and it depends on viewing task 200, which can be obtained from the 
storage for customization information 40. If the task 200 is to "provide fun and 
comfortable experience" then the comfort level could be high (e.g., 90), if the task 
is to "optimize detectability" then the comfort level could be moderate (e.g., 70). 

20 In step 555 a fusion comfort range is calculated for both crossed and 

uncrossed conditions. For this, a corresponding Fusion Probability, FP 9 is 
calculated based on the comfort level, CL: 



FP - k CL x CL c CL 

25 

where kcL and cql are linear scaling constant. In experiments 
conducted by the authors it was found that the constant kcL = 0.84 and ccl - 20.8 
provided good data fit (R2=0.95), and can be implemented in the preferred 
embodiment. Next, the fusion Comfort Range, CR, is calculated based on the 
30 following formula: 
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ln 

CR(crossed) = w reserves xFR(crossed) d + w phoria xP d -- 



' - — 1^ 

FP 



In 

CR{uncrossed) = -w reseives x FR(uncrossed) d + w phoria xP d +- 



(16) 

FP 



(17) 



where w reserves and w p h oria are weighting parameters for 
5 correspondingly fusional reserves and phoria effects; b is a nonlinear parameter of 
a logistic function describing relationships between the fusional range and the 
fusional probability. According to authors' experimental data the values w resen , es - 
0.3; w p horia = 1.0; b = -5 can be used in Equations 16 and 17. 

The obtained comfort ranges can be related to Aim Range of 
1 0 Crossed Disparity (ARCD) or Aim Range of Uncrossed Disparity (ARUD) using 
equations 18 and 19: 

ARCD = 2 x CR(crossed) ^ j ^ 

j 5 ARUD = 2x -CR{uncrossed) 

Fig. 9 shows an illustration plot representing the display 
accommodation plane 780 and optometric data: Near base-out fusional reserve 

20 522, Near base-in fusional reserve 524, Distant base-out fusional reserve 526, 
Distant base-in fusional reserve 528, Near phoria 518, Distant phoria 520 and 
Equal convergence-accommodation demand line 770 are represented in this figure. 
Note that the shaded area 790 represents the range of disparities that one could 
reasonably expect a user to be able to fuse. 

25 A method of obtaining the customization information and defining 

the aim disparity range has now been described. Referring to Fig. 2, the display 
attributes are obtained in step 110 from storage device for display attributes 30. 
The attributes include but are not limited to the following parameters: resolution, 
size of display, point of optical convergence, plane of accommodative focus, and 

30 display refresh rate. 
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The stereo images are obtained in step 120. For imagery scenes 
they can be pairs of images for left and right eye views. For computer graphics the 
3-D model of the scenes and the rendering method (including camera separation 
and focal length) are obtained. 
5 In step 130 a disparity map for a pair of images is obtained. For 

imagery, distance to each pixel may be recorded at the time of capture. If a stereo 
image pair has been captured without depth information, there are several existing 
methods in the prior art to obtain this disparity map, including object matching and 
phase correlation. For scenes that are generated using computer graphics, the 

10 disparity map can be obtained through analytical calculation, given the knowledge 
of scene 3-D model and the rendering method. 

Fig. 10 shows the relationship of the spatial arrangement of a pair of 
stereo cameras that may have been used to capture a real scene or that might have 
been simulated in a 3D graphical environment, the scene depth range, and the 

1 5 corresponding scene disparity range. Two cameras 800 and 8 1 0 are placed side by 
side with a separation distance 870. The converging point of the two cameras is at 
820, which is away from the camera by a distance 880. The scene contains an 
object at the farthest distance 830 and another object at the closest distance 840. In 
the captured scenes, the convergence plane 820 has zero disparity. Between the 

20 camera and the convergence point is the Scene Range of Crossed Disparity 860 

(SRCD). Between the convergence point and the farthest object in the scene is the 
Scene Range of Uncrossed Disparity 850 (SRUD). 

In step 140 a customized disparity map is generated. When images 
are input, this step will generally consist of redefining the convergence point, and 

25 remapping the scene disparity range based on the aim disparity range. Fig. 1 1 
shows the remapping process. On the scene side there is the farthest 
object/maximum uncrossed disparity 830, convergence point/zero disparity 820, 
closest object/maximum crossed disparity 840, Scene Range of Uncrossed 
Disparity (SRUD) 850, and Scene Range of Crossed Disparity (SRCD) 860. On the 

30 aim side there are aim maximum uncrossed disparity 890, aim zero disparity/aim 
convergence point 910, aim maximum crossed disparity 930, Aim Range of 
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Uncrossed Disparity (ARUD) 900, and Aim Range of Crossed Disparity (ARCD) 
920. For this particular case, the overall scene disparity range (SRUD+SRCD) is 
larger than the aim disparity range (ARUD+ARCD), and the ratio of SRUD to 
SRCD is not equal to the ratio of ARUD to ARCD. The first step is to move the 
5 convergence point of the scene by x to fulfill the following requirement: 



(SRUD-x)/(SRCD+x) = ARUD/AUCD (20) 

where x is a signed entity, and a positive value for x means that the convergence 
1 0 point is moved away from the cameras. SRCD, SRUD, ARUD, and ARCD are 
range values, and always have positive values. 

The second step is to rescale the total range of scene disparity so 
that it matches the aim range of disparity. Fig. 12(a) shows an example of linear 
mapping. Here a scale factor K is calculated in equation 21 : 

15 

K*(SRUD+SRCD) = ARUD + AUCD (2 1 ) 



20 



The relationship of Remapped Scene Disparity (RSD) and the 
Original Scene Disparity (OSD), therefore, is: 

RSD = K*(OSD - x) (22) 



where RSD, OSD and x are all signed entities. Positive values represent crossed 
disparity, and negative values uncrossed disparity. In the linear remapping the 

25 disparity is compressed equally throughout the disparity range, and there is no 
distortion of depth in the scene. 

The remapping between Remapped Scene Disparity (RSD) and the 
Original Scene Disparity (OSD) can also take other forms. For example, it can be 
a nonlinear mapping, as shown in Fig. 12(b). It can also take into account the 

30 region of interest (ROI), as shown in Fig. 1 2(c). 
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In a computer graphics environment the step of generating a 
customized disparity map 140 may consist of the location, orientation, focal 
distance, magnification and depth of field characteristics of a pair of cameras that 
are used to emulate actual cameras as if they are capturing the scene. In this 
5 embodiment, the convergence point can most easily be modified by changing the 
location, orientation and focal distance of the cameras to provide a point where the 
two cameras are converged and focused to a point in space. The scene disparity 
range can then be modified by changing parameters such as the separation of the 
cameras or the magnification of the scene to increase or decrease disparities. 

1 0 Alternatively, the depth of field of the cameras may be adjusted to blur near or 
distant objects such that the human brain is unable to interpret the disparity 
information from these near or distant objects. Finally, it is possible to relocate 
objects within the scene, moving them closer to the convergence point to reduce 
disparities or to move them further from the convergence point to increase 

15 disparities. An example of this method is described in U.S. Pat. No. 6,175,379B1. 
Similar methods may be implemented in computer graphics cards such as the 
NVIDIA 3D stereo driver that was described earlier. In these systems, new 
rendering parameters may be determined that are loaded into registers within this 
video card to alter the rendering process as required. 

20 This disparity remapping process described in steps 130, 135, and 

140 may apply to a single stereo image pair, an entire computer graphics scene, or 
an entire video sequence of stereo image pairs. 

After the customized disparity map is generated, stereo image pairs 
will either be rendered within systems employing computer graphic models or re- 

25 rendered for systems employing stereoscopic image data. In a system employing 
computer graphics, the convergence point and the disparity range may be used to 
establish rendering parameters for computer graphics systems. 

In systems employing stereoscopic image information, the scene 
may be re-rendered to remap the disparities of objects in the left or right eye to 

30 match the aim disparity map. Various means for performing this mapping are 
known that either use information from one image to generate a second image or 
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that use information from multiple image frames to generate a second image frame. 
For example, the remapping methods described in, WO2003000122A1 and U.S. 
Pat. No. 4,647,965, can be used in the present invention. In systems, employing 
computer graphics, stereo images may simply be rendered for display. 
5 The remapping and re-rendering process in steps 135, 140 and 150 

can be a dynamic process, and is dependent on the input from sensor 80. When the 
sensor 80 senses a new head position, it means that the viewing distance is 
changed. This would result in a change in the aim disparity range, as depicted in 
Fig. 9. 

10 The final step of the method is to display the images in step 160. 

The images will be displayed using stereoscopic display device 60. 

While this embodiment has been described to provide a customized 
rendering for each individual user, it is also possible to pre-render one or more 
scenes using the process described in Fig. 2 for a number of common user 

15 characteristics. Accordingly, a user is defined as both an individual user of the 
stereoscopic display and also as a cluster of individual users having common 
perceptual characteristics for stereoscopic viewing. One may also pre-select a few 
rendering parameters. In either case, the process applied in Fig. 5 can then be used 
to classify each user into a category for one of the selected groups by determining 

20 which of the pre-selected groups has a visual performance that most closely 

matched the users visual performance. Alternatively, the user may self classify 
themselves using even simpler means of indicating that they belong in a group 
such as indicating if they typically experience eye strain or headaches when using 
stereo displays. Further, under these conditions, the process applied in Fig. 5 can 

25 be simplified to include displaying a single scene or a series of scenes with similar 
depth information that have been rendered with different rendering parameters and 
then providing the user to select the image rendering that is the most useful or 
comfortable to view. Once this image is selected, the set of scenes with this same 
rendering may be presented to the user and/or all future images may be rendered 

30 using similar rendering parameters. 
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The invention has been described with reference to a preferred 
embodiment. However, it will be appreciated that variations and modifications 
be effected by a person of ordinary skill in the art without departing from the 
scope of the invention. 
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